System and method for multiuser interaction with a media computing device with multiple virtual screens and personal area network capability

ABSTRACT

A system for multiuser interaction with a media computing device with multiple virtual screens, comprising: a media computing device; and a plurality of mobile devices. The media computing device sends screen graphics via a virtual screen driver to the mobile devices using a wireless network. Each mobile device displays a virtual screen using screen graphics sent by the media computing device. Touchscreen user interaction events are transmitted from the mobile devices to the media computing device and processed by the operating system thereof.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation-in-part of U.S. patentapplication Ser. No. 15/178,544, titled “MEDIA COMPUTING DEVICE WITHMULTIPLE VIRTUAL SCREENS AND PERSONAL AREA NETWORK CAPABILITY” and filedon Jun. 9, 2016, which claims priority to U.S. provisional patentapplication Ser. No. 62/328,562, titled “MEDIA COMPUTING DEVICE WITHMULTIPLE VIRTUAL SCREENS” filed on Apr. 27, 2016, the entirespecification of each of which is incorporated herein by reference inits entirety. The present application also claims the benefit of, andpriority to, U.S. provisional patent application Ser. No. 62/332,476,titled, “SYSTEM AND METHOD FOR MULTIUSER INTERACTION WITH HDMI-BASEDCOMPUTING DEVICE”, and filed on May 6, 2016, the entire specification ofwhich is incorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION Field of the Art

The disclosure relates to the field of computing devices, and moreparticularly to the field of media computing devices.

Discussion of the State of the Art

Streaming multimedia devices such as ROKU™ or CHROMECAST™ are often usedto play media to a television or other external display with minimalhardware and software features, as a dedicated media consumption devicefor home use. More advanced functions such as gaming or other softwareapplications as may be commonplace in other computing devices, are notgenerally possible or when possible, they “take over” the television asthe primary display and interrupt media consumption. Additionally, suchdevices generally operate directly via the television and any additionaldevices such as multiple-speaker arrangements for multi-channel audioare operate by the television or a separate media “head unit”.

What is needed, is a means to stream media to an external display devicewhile allowing OS and application interaction via a remote, sharedscreen accessible via a user's mobile device, that can connect tomultiple media destination devices for multicast streaming without theneed for an external head unit.

SUMMARY OF THE INVENTION

Accordingly, the inventor has conceived and reduced to practice, in apreferred embodiment of the invention, a system and method for multiuserinteraction with a media computing device with multiple virtual screensand personal area network capability. The following non-limiting summaryof the invention is provided for clarity, and should be construedconsistently with embodiments described in the detailed descriptionbelow.

To address the problem of providing a means to stream media to anexternal display device while allowing wireless operating system (“OS”)and application interaction by one or more users of mobile electronicdevices, the inventors conceived and reduced to practice a device thatmay be inserted into a standard high-definition media interface (“HDMI”)socket of a television, or into a similar media interface such asDisplayPort or other media adapter port in a television or similar mediadevice. The media computing device has a complete operating system suchas ANDROID™, IOS™, LINUX™, or WINDOWS™. It is equipped with one or morewireless network interface circuits, enabling it to communicate via WiFiwith other devices. The device is controlled by users of remote mobiledevices such as smart phones or tablets, via WiFi. The media computingdevice does not necessarily have any built-in user interface, but ratheruses a virtual screen driver to provide full operating system userinterface screens to users of mobile devices. One or more virtualscreens may be operated simultaneously by mobile device users. A mobiledevice user is presented with a typical operating system home screen ordesktop (depending on the operating system), and can interact with thescreen using touchscreen capabilities to perform any actions normallyavailable in the operating system. The home screen, virtual screen, orvirtual desktop appears within an application on the mobile device, thusproviding a “full OS inside an app” modality to the user, with actionstaken by the user affecting what appears on a television.

The delivery of virtual screens to mobile devices from the mediacomputing device may be done by intercepting screen graphics bitmaps(which would normally be sent to a hardware display device forrendering), and transcoding the graphics data into H.264 (for example).The H.264 stream can then be sent in packetized form over a wirelessnetwork to each of the mobile devices. Similarly, user interactionevents received on the mobile devices (for example, touchscreeninteractions) are sent in packetized form over the wireless network tothe media computing device, where they are provided to and interpretedby the operating system as if they were user interaction events arrivingfrom an attached touchscreen. In some use modes, each user sees the sameOS screen on his mobile device. In one of these modes, each user cansimultaneously interact with the OS via his mobile device and the OSuses an interrupt management system (common in operating systems) toensure that each user's interactions are handled in a timely manner. Inanother use mode involving a shared virtual screen, only one user at atime has control of the user interface; users could take turnscontrolling, or users could “seize control” with or without theconcurrence of other users (quasi-competitive control of the shared userinterface could be part of a game concept). In another use mode, eachuser is provided with his own virtual screen. For example, manyoperating systems provide for multiple virtual desktops, so that a usermay use different applications on different desktops. In the use modeenvisioned, each user interacts with the operating system using his ownlogical/virtual desktop, which is provided wirelessly as before.

Using multiple virtual screens streamed to mobile devices enables avariety of use cases for users, including new ways to enjoy video games,watch video content, or listen to audio both actively and passively inthe background. Multiple users wishing to play a video game may usetheir smartphones, tablets, or other mobile devices as both displayscreens and controllers for new multiplayer game types, such as multipleusers cooperating to control the bridge of a ship in a simulation game.Each player sees their respective game environment on their mobiledevice screen, such as their particular control station or immediatesurroundings (for example, using hardware sensors in the mobile deviceto enable a user to “look around” by moving their device). On thetelevision or similar media device, an overview of the game is shown,such as a map or a view of the ship's entire bridge, showing all playersand the game environment in a single view. As players interact withtheir devices to react to game events, interact with each other throughthe game, and take in-game actions, the various game views being shownare updated in response to these interactions to reflect the ongoinggame state. If a player moves about the room, their mobile devicecontinues to act as their localized input and output device for thegame, allowing them to continue playing, for example, even when theyleave the room temporarily such as to use the restroom, prepare food,answer the door or phone, etc.

Using streamed virtual screens for presenting information and receivinginteraction, it becomes possible to enable individual media consumptionfor each user by providing their specific media content to their deviceand receiving their interactions such as media selection or controlinteractions, without affecting other users or the media being shown ona connected television or other media device. Additional connecteddevices such as networked speakers in a multi-room audio arrangement maybe utilized in conjunction with streamed virtual screens to accommodatemore complex use cases, such as position-based multi-room audio. As auser moves about within a multi-room audio setup, their mobile device'sposition tracking (such as using network signal or GPS sensors) may beprovided to the media computing device, which then directs the connectedaudio devices accordingly so that the user is always listening to theirspecific audio selection via the nearest speaker or other device (forexample, using integrated speakers on a connected television or computermonitor when available). When multiple users move about, media streamingis continuously updated so that each user's personalized mediaselections are sent to the appropriate audio device, and conflicts maybe resolved using preconfigured handling rules such as “first come,first served” or a user hierarchy such as giving priority to the userthat owns the media computing device or is hosting the gathering.

Using personal area network functionality, a user may view and interactwith their connected internet-of-things devices using a media computingdevice as a hardware “hub”, removing the need for multiple hubs orbridges to connect devices from different manufacturers or that usedifferent protocols. This also enables a convenient, unified interfacefor interacting with these devices while the user is already connectedto the media computing device, removing the need for multiple softwareapplications or additional configuration, and it also enablesintegration of these devices with other functions of the media computingdevice, such as to cause device operations to occur based on mediaplayback or complex rules configured within the media computing devicethat may not be possible within an IoT device or hub normally.

By offering a wide-area network interface for use, a media computingdevice can serve as a backup Internet connection when a local network isunavailable, enabling full functionality regardless of network setup orstate. This allows a user to utilize the full functionality of the mediacomputing device while traveling or during periods of connectivity losson their local network, and enables the use of the media computingdevice as a WAN access point or a fallback modem for a home network,removing the need for separate devices to perform those roles.

According to a preferred embodiment of the invention, a system formultiuser interaction with a media computing device with multiplevirtual screens is disclosed, the system comprising: a media computingdevice comprising a processor, a memory, a high-definition multimediainterface adapter, a wireless network interface, a virtual screendriver, and an operating system; and a plurality of mobile devices, eachcomprising at least a processor, a memory, a touchscreen display, and awireless network interface. According to the embodiment, the mediacomputing device sends screen graphics via the virtual screen driver tothe plurality of mobile devices using the wireless network; each of theplurality of mobile devices displays a virtual screen of the operatingsystem using the screen graphics sent by the media computing device; andtouchscreen user interaction events are transmitted from the pluralityof mobile devices to the media computing device and processed by theoperating system thereof.

According to another preferred embodiment of the invention, a method formanaging a multiplayer video game using multiuser interaction with amedia computing device with multiple virtual screens, comprising thesteps of: running a game application in user space of an operatingsystem of a media computing device; sending, using a high-definitionmultimedia interface adapter of the media computing device, an overviewgame display for a game application to a high-definition multimediadisplay device physically connected to the high-definition multimediainterface adapter of the media computing device; presenting, usingscreen graphics generated by a virtual screen driver of the mediacomputing device, a virtual screen comprising a player-specific gamedisplay on each of a plurality of connected mobile devices; receiving auser interaction event from a first mobile device at the media computingdevice; processing, using the game application, the user interactionevent to determine whether each respective player-specific game displayor the overview game display is affected by the user interaction event;and updating the affected game displays.

According to a further preferred embodiment of the invention, a methodfor handling multi-user audio using multiuser interaction with a mediacomputing device, comprising the steps of: transmitting screen graphicsgenerated by a virtual screen driver of the media computing device, thescreen graphics corresponding to a virtual screen comprising a mediacontrol interface, to a plurality of connected mobile devices;receiving, at the media computing device, a plurality of userinteraction events from the plurality of mobile devices; selecting aplurality of audio playback devices based on a first user interactionevent; selecting media content for playback based on a second userinteraction event; and wirelessly transmitting, from the media computingdevice, audio from the selected media content for playback at theplurality of audio playback devices. According to further embodiments ofthe invention, the method further comprises the step of sending, using ahigh-definition multimedia interface adapter of a media computingdevice, high-definition video and a plurality of audio channelsassociated with the media content to a high-definition multimediadisplay device physically connected to the high-definition multimediainterface adapter of the media computing device. In yet anotherembodiment, playback of the video and audio sent to the high-definitionmultimedia display device is synchronized with playback of the audiotransmitted to the plurality of audio playback devices.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The accompanying drawings illustrate several embodiments of theinvention and, together with the description, serve to explain theprinciples of the invention according to the embodiments. It will beappreciated by one skilled in the art that the particular embodimentsillustrated in the drawings are merely exemplary, and are not to beconsidered as limiting of the scope of the invention or the claimsherein in any way.

FIG. 1 is a block diagram illustrating an exemplary system architecturefor a media computing device with multiple virtual screens and personalarea network capability, according to a preferred embodiment of theinvention.

FIG. 2 is a block diagram illustrating an alternate exemplary systemarchitecture for a media computing device with multiple virtual screensand personal area network capability, according to a preferredembodiment of the invention

FIG. 3 is a flow diagram illustrating an exemplary method for multiuserinteraction with a media computing device with multiple virtual screensand personal area network capability, illustrating the use of multipleusers each having a unique screen instance, according to a preferredembodiment of the invention.

FIG. 4 is a flow diagram illustrating an exemplary method for multiuserinteraction with a media computing device with multiple virtual screensand personal area network capability, illustrating the use of multipleusers sharing a single screen instance, according to a preferredembodiment of the invention.

FIG. 5 is a flow diagram illustrating an exemplary use case formultiuser interaction with a media computing device with multiplevirtual screens and personal area network capability, illustratinghandling for a multiplayer video game.

FIG. 6 is a flow diagram illustrating an exemplary use case formultiuser interaction with a media computing device with multiplevirtual screens and personal area network capability, illustratinghandling for multi-user audio listening.

FIG. 7 is a block diagram illustrating an exemplary hardwarearchitecture of a computing device used in an embodiment of theinvention.

FIG. 8 is a block diagram illustrating an exemplary logical architecturefor a client device, according to an embodiment of the invention.

FIG. 9 is a block diagram showing an exemplary architectural arrangementof clients, servers, and external services, according to an embodimentof the invention.

FIG. 10 is another block diagram illustrating an exemplary hardwarearchitecture of a computing device used in various embodiments of theinvention.

FIG. 11 is an illustration of an exemplary device arrangement utilizinga media computing device with multiple virtual screens and personal areanetwork capability, illustrating the connection the media computingdevice to a television and speakers, and wireless presentation ofmultiple virtual screens to mobile devices.

FIG. 12 is a block diagram illustrating an alternate exemplary systemarchitecture for a media computing device with multiple virtual screensand personal area network capability, illustrating the use of a PANcontroller to function as a home automation hub according to a preferredembodiment of the invention.

FIG. 13 is a flow diagram illustrating an exemplary method for utilizinga PAN controller to operate a media computing device with multiplevirtual screens and personal area network capability as a homeautomation hub, according to a preferred embodiment of the invention.

FIG. 14 is a diagram illustrating an exemplary user interface forinteracting with PAN devices via a media computing device with multiplevirtual screens and personal area network capability, according to apreferred embodiment of the invention.

FIG. 15 is a flow diagram illustrating an exemplary method for multiuserinteraction with PAN devices via a media computing device with multiplevirtual screens and personal area network capability, according to apreferred embodiment of the invention.

FIG. 16 is a flow diagram illustrating an exemplary method for using amedia computing device with multiple virtual screens and personal areanetwork capability as a mobile wireless access point.

DETAILED DESCRIPTION

The inventor has conceived, and reduced to practice, in a preferredembodiment of the invention, a system and method for multiuserinteraction with a media computing device with multiple virtual screensand personal area network capability.

One or more different inventions may be described in the presentapplication. Further, for one or more of the inventions describedherein, numerous alternative embodiments may be described; it should beappreciated that these are presented for illustrative purposes only andare not limiting of the inventions contained herein or the claimspresented herein in any way. One or more of the inventions may be widelyapplicable to numerous embodiments, as may be readily apparent from thedisclosure. In general, embodiments are described in sufficient detailto enable those skilled in the art to practice one or more of theinventions, and it should be appreciated that other embodiments may beutilized and that structural, logical, software, electrical and otherchanges may be made without departing from the scope of the particularinventions. Accordingly, one skilled in the art will recognize that oneor more of the inventions may be practiced with various modificationsand alterations. Particular features of one or more of the inventionsdescribed herein may be described with reference to one or moreparticular embodiments or figures that form a part of the presentdisclosure, and in which are shown, by way of illustration, specificembodiments of one or more of the inventions. It should be appreciated,however, that such features are not limited to usage in the one or moreparticular embodiments or figures with reference to which they aredescribed. The present disclosure is neither a literal description ofall embodiments of one or more of the inventions nor a listing offeatures of one or more of the inventions that must be present in allembodiments.

Headings of sections provided in this patent application and the titleof this patent application are for convenience only, and are not to betaken as limiting the disclosure in any way.

Devices that are in communication with each other need not be incontinuous communication with each other, unless expressly specifiedotherwise. In addition, devices that are in communication with eachother may communicate directly or indirectly through one or morecommunication means or intermediaries, logical or physical.

A description of an embodiment with several components in communicationwith each other does not imply that all such components are required. Tothe contrary, a variety of optional components may be described toillustrate a wide variety of possible embodiments of one or more of theinventions and in order to more fully illustrate one or more aspects ofthe inventions. Similarly, although process steps, method steps,algorithms or the like may be described in a sequential order, suchprocesses, methods and algorithms may generally be configured to work inalternate orders, unless specifically stated to the contrary. In otherwords, any sequence or order of steps that may be described in thispatent application does not, in and of itself, indicate a requirementthat the steps be performed in that order. The steps of describedprocesses may be performed in any order practical. Further, some stepsmay be performed simultaneously despite being described or implied asoccurring non-simultaneously (e.g., because one step is described afterthe other step). Moreover, the illustration of a process by itsdepiction in a drawing does not imply that the illustrated process isexclusive of other variations and modifications thereto, does not implythat the illustrated process or any of its steps are necessary to one ormore of the invention(s), and does not imply that the illustratedprocess is preferred. Also, steps are generally described once perembodiment, but this does not mean they must occur once, or that theymay only occur once each time a process, method, or algorithm is carriedout or executed. Some steps may be omitted in some embodiments or someoccurrences, or some steps may be executed more than once in a givenembodiment or occurrence.

When a single device or article is described herein, it will be readilyapparent that more than one device or article may be used in place of asingle device or article. Similarly, where more than one device orarticle is described herein, it will be readily apparent that a singledevice or article may be used in place of the more than one device orarticle.

The functionality or the features of a device may be alternativelyembodied by one or more other devices that are not explicitly describedas having such functionality or features. Thus, other embodiments of oneor more of the inventions need not include the device itself.

Techniques and mechanisms described or referenced herein will sometimesbe described in singular form for clarity. However, it should beappreciated that particular embodiments may include multiple iterationsof a technique or multiple instantiations of a mechanism unless notedotherwise. Process descriptions or blocks in figures should beunderstood as representing modules, segments, or portions of code whichinclude one or more executable instructions for implementing specificlogical functions or steps in the process. Alternate implementations areincluded within the scope of embodiments of the present invention inwhich, for example, functions may be executed out of order from thatshown or discussed, including substantially concurrently or in reverseorder, depending on the functionality involved, as would be understoodby those having ordinary skill in the art.

Conceptual Architecture

FIG. 1 is a block diagram illustrating an exemplary system architecture100 for a media computing device with multiple virtual screens andpersonal area network capability, according to a preferred embodiment ofthe invention. According to the embodiment, a media computing device 110may comprise a system-on-a-chip 115 further comprising a memory 11 and aprocessor 12 (as described below, referring to FIG. 4), configured tooperate without integral hardware display or input devices (such as, forexample, a computer monitor or keyboard as are commonly present incomputing devices), and configured to operate a software operatingsystem 116, for example including (but not limited to) ANDROID™, APPLEIOS™, WINDOWS™, and various forms of LINUX™. OS 116 may comprise a mediamanagement subsystem 117 that provides media content for viewing via anHDMI display device 130 connected via HDMI interface 105 and controlledby HDMI controller 101, such as a television or computer monitor, and avirtual screen driver 118 that may operate a plurality of virtual(logical) interactive software interfaces (“virtual screens”) and maypresent at least a portion of these screens for interaction via aplurality of mobile devices 120 a-n communicating via a plurality ofwireless network interfaces 102 a-n. Operating system 116 may generallyhave a “home screen” or similar primary environment that may be used bya user to interact with various hardware or software features andfunctions of the media computing device 110. Alternatively, OS 116 mayprovide a plurality of logical desktops or other virtual screens tousers via mobile devices 120 a-n. To facilitate user interaction,virtual screen driver 118 may listen for connections via a network orphysical connection from a plurality of mobile devices 120 a-n (such as,for example including but not limited to, a tablet computing device orsmartphone), and may present a virtual screen to a device afterconnecting. A user may then interact with his mobile device normallyusing whatever means are available according to the particularconfiguration of the mobile device, and this interaction may be providedto virtual screen driver 118 for translation and delivery to OS 116. Inthis manner, media may be presented on an external display device 130such as a high-definition television (the device of the inventiontypically being inserted into an HDMI port of the television, andcontrolling the television using HDMI interface 105), while usersinteract with an operating system via virtual screen driver 118,allowing users to interact with software applications. This allows usersto perform actions such as installing programs, playing games, modifyingmedia playback configurations, selecting media for presenting toexternal display device 130, or performing administrative tasks.

It should be noted that, while HDMI is used in the example of FIG. 1 andis discussed throughout, other similar media interfaces may be used inplace of HDMI, according to the invention. For example, DISPLAYPORT™interfaces may be used; in such embodiments, HDMI controller 101 andHDMI interface 105 would be instead a DISPLAYPORT™ controller 101 and aDISPLAYPORT™ interface 105; similarly, other newly emerginghigh-definition media interfaces may be used in various embodiments ofthe invention.

According to various alternative arrangements, virtual screen driver 118may connect via physical connections such as a USB cable (or other cableconnections), for example to connect to a mobile device 120 a-n withparticular hardware capabilities, such as a personal computing devicethat may have a USB port but no active network connection. Wirelessnetwork interface 102 a-n may use a variety of different technologies orprotocols, such as using various frequency bands or channels for WiFiconnectivity, BLUETOOTH™, cellular radios, or other network connectiontypes, to connect mobile devices 120 a-n to media computing device 110.

Virtual screen driver 118 may operate multiple connections for multiplemobile devices 120 a-n, for example so that more than one user mayinteract with the operating system of media computing device 110.According to various arrangements, each mobile device 120 a-n may bepresented a separate “instance”, or copy, of an operating system homescreen or other virtual screen, so that each user may have access to theoperating system for interaction without being affected by theactivities of other users that may be connected. In some arrangements,user access may be restricted such as to provide read-only access tosome portions of an operating system, or to restrict softwareapplications or functions based on configured criteria such as mobiledevice or connection type, or using stored software-based whitelistingor blacklisting behavior. In other arrangements, multiple mobile devices120 a-n may be presented with the same instance of a home screen, sothat multiple users may interact collaboratively such as to worktogether while editing stored files, or to interact with a shared screenin a gaming application, or other shared-use cases. When multiple usersare connected to a single virtual screen driver 118, congestion may bemanaged by operating a “safe mode” or minimal interface for new users,such as when system resources are limited or after a set number ofconcurrent users has been reached (for example, providing a fullinterface to the first four users and a stripped-down interface to anyusers after that). For example, a new user connecting may be presentedwith a notification that the maximum number of connections has beenreached, or that resources are limited, and be prompted to abort orcontinue with limited functionality. A limited interface may comprise,for example, basic remote control functionality to direct the operationof media management subsystem 117 remotely, for example features such as“next track”, “play/pause”, or volume controls. In this manner,additional users may still be able to control media being viewed ondisplay device 130, without requiring full OS interaction.

Operating system 116 may operate a variety of software applications, forexample from an app store such as GOOGLE PLAY™ or similar, according tothe nature of a particular operating system (for example, apps fromGOOGLE PLAY™ may be natively supported on any ANDROID™ operating system,or apps from the Apple App Store on an IOS™ operating system). In thismanner users may utilize familiar software and compatibility may be“future-proofed” by utilizing existing repositories and standards toensure forward-compatibility with new software developments. Whilerunning software applications, multiple users may connect and interactvia their mobile devices 120 a-n, for example for multiple users to playa multiplayer video game application. According to various arrangements,multiple users may share a single screen during interaction, eachviewing and interacting with the same instance of the operating systeminterface, or users may each view and interact with a separate instance,or various combinations thereof. Using multiple screen instances in thismanner may be used to provide complex interaction environments formultiple users, for example for a video game application wherein adisplay device 130 may present an overview of a game environment (suchas a game board, map, aircraft cockpit, ship bridge, or other group oroverview type environment according to the particular application or usecase), while each mobile device 120 a-n may present a user-specific viewsuch as that particular player's game information (for example,character or player statistics) or their particular game environment(for example, their specific character's view of a game world, or asmall-scale map centered on their location within a larger overview mapshown on display device 130, or their specific controls or station in acockpit or bridge environment).

FIG. 2 is a block diagram illustrating an alternate exemplary systemarchitecture 200 for a media computing device with multiple virtualscreens and personal area network capability, according to a preferredembodiment of the invention. According to the embodiment, a mediamanagement subsystem 117 operating on a media computing device 110 mayconnect to multiple destination devices including (for example) aplurality of HDMI display devices 130 a-n such as a computer monitor ortelevision, or a plurality of audio devices 201 a-n, such as speakersconnected via physical connections such as SP/DIF optical audio cables,or connected via network connections such as WiFi or BLUETOOTH™.Multiple HDMI output devices may be used via an HDMI splitter 205, whichenables a single HDMI output from HDMI interface 105 to be sent tomultiple HDMI devices, each of which may use some or all of the mediacontent sent from HDMI controller 101 via HDMI interface 105 and HDMIsplitter 205. For example, a multi-speaker home theater setup may beconnected to a single media management subsystem 117, which may thenbroadcast media in a multicast fashion to some or all connected devicesaccording to various arrangements. In this manner, it may be appreciatedthat a media computing device 110 may be fully compatible with a widevariety of device arrangements and hardware capabilities, and may easilyintegrate into any existing media configuration.

According to an exemplary arrangement, audio may be multicast withminimal latency for synchronized playback at multiple audio devices 201a-n, for example by using a wireless interface 102 a-n to function as awireless access point (WAP) in an infrastructure mode, wherein audiodevices 201 a-n may connect directly to the WAP rather than connectingto a home router or other access point (thereby reducing the number of“hops” or network nodes the audio must traverse between source anddestination).

According to another exemplary arrangement, audio may be streamed tomultiple audio devices 201 a-n wirelessly, and selection of audiodevices 201 a-n for streaming may be updated according to variousfactors such as network performance, media attributes (such as streamingto a subwoofer device for bass-heavy audio tracks, but stopping thestream to that device if the audio style changes) or mobile devicelocation. For example, a user may have multiple audio devices 201 a-nplaced in different areas or rooms. As the user moves about, media maybe streamed to the nearest audio device(s) based on location or networkinformation from their mobile device (such as using WiFi or BLUETOOTH™signal strength or triangulation). In such a use mode, each connecteduser may select their own media, optionally from a streaming third-partysource such as that user's SOUNDCLOUD™ or PANDORA™ account, and theuser-specific media selection will “follow” the user by their locationso that each user only hears their own media. When multiple users enterthe same area, they may optionally be prompted for new media selections,for example incorporating each user's preferences or media collection sothat a group may select music that appeals to everyone or setup aplaylist using media from each user's respective library so thateveryone gets a chance to hear their selection. Additionally, whileaudio is being streamed to multiple devices in this manner, video(optionally with its own audio) may be shown on a connected HDMI displaydevice 130 a-n, for example a music video corresponding to the nearestuser's music selection, or a separate media selection entirely. Thiscontent may optionally be based on selection from users in the vicinityof the display device 130 a-n, in a manner similar to each userselecting media to hear from their nearest audio devices.

FIG. 11 is an illustration of an exemplary device arrangement utilizinga media computing device with multiple virtual screens and personal areanetwork capability, illustrating the connection the media computingdevice 110 to a television 130 and speakers 201 a-n, and wirelesspresentation of multiple virtual screens to mobile devices 120 a-n.According to the embodiment, a media computing device 110 may beconnected to an appropriate hardware port (for example, an HDMI videoport) on the back 801 connection panel of a television 130 or similarvideo device (for example, in some arrangements a computer displaymonitor, projector, or other video device may be used). When connectedin this fashion, media computing device 110 may also draw power foroperation from television 130, removing the need for any additionalcables or connections. television 130 may also be connected to aplurality of external hardware speakers 201 a-n, as is common in homemedia arrangements where the television 130 displays video content andspeakers 201 a-n are used to playback corresponding audio content duringviewing. Media computing device 110 may be further connected viawireless network connections (using wireless network interfaces 102 a-n,as described above with reference to FIG. 1) to a plurality of usermobile devices 120 a-n, for example a plurality of smartphone devices asshown (however it should be appreciated that various types of device maybe used, for example tablet computing devices or laptop personalcomputers, or any other device capable of communicating with mediacomputing device 110 via a wireless network connection). While connectedin this manner, media computing device 110 may present a plurality ofdifferent virtual screens to mobile devices 120 a-n using a virtualscreen driver 118, for example so that different users may interact withmedia computing device 110 independently. For example, one user may playa video game on their mobile device 120 a, while another user is viewinga home screen (or other form of primary or root environment for anoperating system operating on media computing device 110, as describedpreviously with reference to FIG. 1) on their device 120 b, while athird user may be viewing video or other media content on their device120 n. While users are viewing and interacting via their respectivevirtual screens in this manner, media computing device 110 may continuedisplaying video content via the connected television 130 and playingaudio via speakers 201 a-n, so that media playback may continueuninterrupted regardless of user interaction and operations via theirrespective virtual screens.

FIG. 12 is a block diagram illustrating an alternate exemplary systemarchitecture for a media computing device with multiple virtual screensand personal area network capability, illustrating the use of a PANcontroller 910 to function as a home automation hub according to apreferred embodiment of the invention.

According to the embodiment, wireless interfaces 102 a-n may comprise avariety of wireless networking or direct-communication protocols ortechnologies, for example including a local area network (LAN) interface102 a, BLUETOOTH™ interface 102 b, wide-area network (WAN) interface 102c such as an LTE modem for connection to a wide-area network 901 such asa cellular network (for example, to enable Internet access and mediastreaming during travel use, such as plugged into an automobileentertainment console, or for use as a cellular modem or hotspot, or toenable interaction with connected IoT devices 140 a-n from outside thelocal network), and personal area network (PAN) interface 102 nconfigured for connecting to a plurality of network-capableinternet-of-thing (IoT) devices 920 a-n across short distances usinglow-power communications, such as ZigBee, Z-WAVE™, or INSTEON™ (forexample). PAN interface 102 n may be directed by a PAN controller 910,for example a ZigBee Coordinator that may form the root of a ZigBeenetwork of IoT devices 920 a-n. In a ZigBee network, multiple devices920 a-n may be controlled by a single ZigBee Coordinator, whichfunctions as the root node for the network structure. Each of aplurality of additional devices 920 a-n may operate as a ZigBee Routeror a ZigBee End Device, according to their function or position in thenetwork structure. A ZigBee Router may act as an intermediate routerbetween the Coordinator and a plurality of additional End Devices,relaying communication to extend the network (ZigBee directcommunication is generally limited to 10 to 20 meters) or to providelogical organization to devices in the network (such as having a routerfor a particular type of device, or a particular physical area). In thismanner, PAN controller 910 may operate as an IoT hub, removing the needfor a separate device to manage IoT devices and protocols and providingnative connectivity to a variety of protocols and technologies whilemanaging their functions and operation.

PAN communication methods may be used to enable connections to aplurality of IoT devices 920 a-n, such as including (but not limited to)smart home hubs, light bulbs or switches, power outlets, or connectedappliances. According to the nature of specific devices or communicationmethods, devices may connect to one another to form a mesh network andovercome communication limitations (for example, ZigBee generally has anoperating range of 10 to 20 meters, which may not cover all connecteddevices in a building or other environment) or to use a central hubdevice such as home automation hubs produced by SMARTTHINGS™ or similar,to connect multiple devices with varying capabilities and expose theirrespective functionalities via a single unified connection. In thismanner, a media computing device 110 may connect to a hub or otherdevice 920 a and communicate via that device to additional connecteddevices 920 n, enabling connectivity and operation across a wide rangeand variety of communication protocols. Additionally, by utilizing a WANinterface 102 b, some arrangements may provide interaction withconnected IoT devices 920 a-n from outside the local network, forexample so that a user may configure or manually direct devices whilethey are away from their home or office. For example, a user may be ableto adjust their connected thermostat or air conditioning device so thatwhen they return home their house will already be at their preferredtemperature, or they may activate a connected camera to check on theirhome's security, or they may receive notifications from devices such assecurity devices or environment sensors. This further removes the needfor extraneous devices by enabling a media computing device 110 tofunction not only as an IoT hub but also as a cellular modem, acting asa bridge between the user's local network and the Internet for remoteaccess and additional functionality. For example, another function thatmay be provided via a WAN 901 may be streaming media from externalsources such as YOUTUBE™ or other media providers, enabling a user toconnect to the Internet and utilize these services while traveling orwhen no local network may be available. This may also be used as aredundancy failsafe, for example if a local connection is experiencingdifficulties such as network congestion or a hardware failure. In such asituation, a WAN interface 102 b may connect directly to WAN 901 so thatoperation may continue uninterrupted until the LAN 102 a connectivity isrestored.

By connecting via wireless interfaces 102 a-n to IoT devices 920 a-n, amedia computing device 110 may interact with and direct the operation ofsuch devices either automatically or based on received user input (suchas from a mobile device 120 a-n). For example, during video playback aplurality of IoT lights 920 a-n (such as network-connected LED bulbs orlight switches controlling non-connected traditional light bulbs) may bedirected to alter their intensity or color based on the video framesbeing shown, to enhance the viewing experience or to reduce eyestrain.In another example, a user interacting with media computing device 110via their mobile device 120 a-n may manually control the operation ofavailable IoT devices 920 a-n via the media computing device 110,without the need for an external software application or controlinterface. In this manner, media computing device 110 may function as anIoT hub device, connecting a variety of IoT devices 920 a-n anddirecting their operation or presenting their functions and informationin a unified manner.

Detailed Description of Exemplary Embodiments

FIG. 3 is a flow diagram illustrating an exemplary method 300 formultiuser interaction with a media computing device with multiplevirtual screens and personal area network capability, illustrating theuse of multiple users each having a unique screen instance, according toa preferred embodiment of the invention. In an initial step 301, a usermay connect to a virtual screen driver 118 operating on a mediacomputing device 110. In a next step 302, the virtual screen driver 118may check to verify whether a configured limit has been reached, forexample a performance threshold that specifies a media quality orsimilar performance-based limit, or a configured maximum number ofusers, or a resource usage threshold such as based on memory orprocessor load, or other limit. If a limit is configured and has beenreached or exceeded, the new user may be notified 303 and offered areduced interface 304 such as a TV remote control interface to providebasic functionality for controlling media streaming without impactingmedia quality or resource usage. If no limit is configured, or aconfigured limit has not been reached, the virtual screen driver 118 maycreate a new instance of an operating system screen 305 such as a homescreen or desktop (as may vary according to the specific nature orconfiguration of an operating system 116, and present it to the user forinteraction via their mobile device 120. When an interaction is receivedfrom a connected user 306, the virtual screen driver 118 may first checkto verify whether a user interface or operating system element involvedin the interaction (for example, a file the user is trying to open, oran interface window or icon they are attempting to interact with ormanipulate) is locked, indicating that another user is already using theresource (for example, if a user tries to reposition a window that isalready being repositioned by another user, such as a “click and drag”conflict that may be possible in some operating systems). If an elementor resource is locked, the requesting user may be notified 308 andoffered a prompt to “break in” to the resource 309, which notifies theuser currently using and locking the resource 310, requesting access. Ifthe locking user denies access 311, the requested operation is denied312. If the locking user grants access 311, or if there is no lock onthe resource, the operation is performed 313.

In an instantiated-screen usage mode, each connected user may view adistinct instance or copy of an OS screen, such as an applicationscreen, home screen, or desktop. Each user may view and interact with OSfeatures such as applications or files, without being affected by otherconnected users (and may not even know how many other users areconnected, if any). Such an operation mode may be useful, for example,for enabling users to work on files or documents separately using theshared computing resources of the media computing device, whilesimultaneously viewing media on a connected display device.

FIG. 4 is a flow diagram illustrating an exemplary method 400 formultiuser interaction with a media computing device with multiplevirtual screens and personal area network capability, illustrating theuse of multiple users sharing a single screen instance, according to apreferred embodiment of the invention. In an initial step 401, a usermay connect to a virtual screen driver 118 operating on a mediacomputing device 110. In a next step 402, the virtual screen driver 118may then present the new user with a virtual screen corresponding to thecurrent state of an operating system screen (such as a home screen,desktop, or application screen), for example showing any runningapplications or open files, and optionally indicating other users'interactions (for example, showing a cursor or other indicator for eachuser, indicating their current or most recent interaction). When a newinteraction is received 403, the virtual screen driver 118 may check tosee if any involved interface elements or resources are locked 404,indicating that another user is already interacting with them. If nolocks are found, the interaction is processed and operation continuesuninterrupted 409.

If a resource is locked, the virtual screen driver 118 may then checkfor a configured operation mode for resolving interaction conflicts 405.If the operating mode is configured to resolve interaction conflictsusing interrupt-based handling, an interrupt request may be generated407, unlocking the element and interrupting or cancelling any operationthat the locking user was performing, such as to “drop” a window theuser was in the middle of “dragging and dropping” as part of a userinterface interaction, or removing control of a shared keyboard or otherinteraction device and then completing the interaction received from theinterrupting user 409. When a user is interrupted, a notification mayoptionally be presented on their mobile device so they are aware of theevent. Interrupts may be processed fluidly enough that the users may noteven be aware they are happening, for example if a user is interruptedwhile dragging an interface window, they may only see that another userhas clicked and is now dragging the window in another direction, andthey may then click and drag again if they wish, all occurring fluidlyas it would normally in a single-user environment.

If the conflict resolution mode is instead configured not to useinterrupts and to maintain a “single-seat control” operation scheme, theuser currently interacting with and locking an element maintains control(optionally being notified of the attempted interaction), and the newinteraction is cancelled or ignored 406. In a single-seat operationmode, only a single user may control a particular element or resource ata time, and new interactions may not be processed for an element orresource until a current interaction is completed (thus relinquishingcontrol). This may be thought of as a user “driving” an interaction orelement, and other users being “passengers” until they are finished.Once control is relinquished, other users may then “take over” andinteract with the now-unlocked element or resource.

In a single-screen usage mode, each connected user interacts with thesame instance or copy of an operating system, potentially interactingwith or affecting other connected users. While interacting with the sameinstance of the operating system, users may be able to view the samescreen simultaneously, such as to view the same file or document forcollaborative work, or they may view different information on theirrespective mobile devices such as (for example) separate applications orfiles. An exemplary use for this operation mode may be a multiplayervideo game, where each user is presented with their own personal view ofthe game, and an overview or admin view is presented on a connecteddisplay device. For example, in a first-person shooter type game, eachplayer may be shown their own character's information such as health orammunition, or a local map based on their character's immediatevicinity. On a connected display device, an overview map of the entireplaying field may be shown, or global game information such as a scorelist for all players. Another exemplary use may be a ship (or spaceship)simulation, wherein the connected display device shows an overview of aship's bridge, reflecting current in-game conditions such as (forexample) player characters at their stations, weather or lightingeffects, other vehicles or characters outside the bridge visible throughwindows or screens, or other such game events. Each player may then beshown their specific station or controls on their mobile device, so thateach player may assume the role of a crew member on the bridge,operating their specific set of controls and systems without seeing orbeing able to interact with another player's controls or systems. Asplayers interact with their specific screen instance, the effects may beshown on the overview page (such as a player toggling the bridgelighting, for example), and optionally on other players' screeninstances where appropriate.

FIG. 5 is a flow diagram illustrating an exemplary use case 500 formultiuser interaction with a media computing device with multiplevirtual screens and personal area network capability, illustratinghandling for a multiplayer video game. In an initial step 501, a primaryor overview display for a game may be shown on a connected television orother media display device (such as a DisplayPort computer monitor,projector, or other display device), for example showing a viewoverlooking the bridge of a ship in a simulation game, or a map orsimilar overview for a game overall (that is, not specific to aparticular player or device). Virtual screens may then be presented toeach connected player 502, showing that player's respective view of thegame such as their specific control station on a ship's bridge in asimulation game, or their character information in a roleplaying game,or their specific POV within a game environment, or otherplayer-specific display. When a player interacts with their mobiledevice 503, this interaction is capture by the virtual screen for use byvirtual screen driver 118, which may then process the interaction. Whileprocessing the player's interaction, virtual screen driver 118 may checkto see whether the interaction results would affect the overview display504, and if so update the view being presented on the television 504 a.Virtual screen driver 118 may then check to determine whether theresults affect other players 505, and if so it updates the view for eachaffected player, and if not it updates the interacting player only (forexample, to confirm their action).

In this manner, all affected views and displays are updated when actionstake place, for example to show players the effects of interacting witheach other while masking those interactions from uninvolved players, orto show the effects of player actions in an overview display. In a shipbridge simulation example, as players take action on their respectivecontrol consoles within the bridge, the results may be reflected in theoverview display such as altering lighting or other simulated systemsoperating on the ship's bridge, or showing the environment changingoutside the bridge as seen through windows or displays such as to showthe effects of navigating the ship or engaging in combat. In aroleplaying game, a map may be updated as individual players explore,and interactions between players or between a player and a non-playercharacter may be shown to the involved players only, simulating theeffects of participating in private conversations within the gameenvironment and encouraging players to cooperate with each other in newways to share, conceal, or manipulate information obtained.

FIG. 6 is a flow diagram illustrating an exemplary use case 600 formultiuser interaction with a media computing device with multiplevirtual screens and personal area network capability, illustratinghandling for multi-user audio listening. In an initial step 601, avirtual screen may be presented to a connected user's mobile device, forexample a media player interface allowing the user to select or providemedia for use, control the manner of playback by manipulating volume orplayback speed, or select a particular connected audio device for use.When an interaction is received via the virtual screen 602, virtualscreen driver 118 may check to determine whether the user selected aspecific audio device 603. If no audio device was specific, one may beautomatically selected 603 a, for example based on the user's proximityor relative network signal to select the audio device that will provideoptimal playback. If the user selected a particular audio device, mediamay be presented via the selected device 604. When a new interaction isreceived 605, virtual screen driver 118 may check the contents of theinteraction again. If the interaction includes a change in selectedmedia 605 b, the new media selection may be received 602 and processedas operation continues.

If the interaction includes a change in audio device selection 605 a,media may be presented to the newly-selected device 604 and playbackhalted or quieted at the previous device. Device selection may occur dueto the user specifically selecting a new device via manual interaction,or it may be partially or fully automated based on information fromhardware sensors or a network interface on the mobile device, which maybe provided as an interaction without manual operation by the user. Forexample, as the user moves about their location information may beupdated and sent as a new interaction to automatically select a newaudio device that is closer to the user's new position than thepreviously-selected device, or that has preferable network signalquality. In this manner, the user may move about freely while stilllistening to their media, as playback “follows” them by updating thedevices in use. By muting or stopping playback at previous devices andplaying only through the current “ideal” device, multiple users may moveabout and listen to their own media each through their respective audiodevice. Alternately, by varying the playback volume instead ofcompletely halting playback at previous devices, audio may smoothlytransition as the user moves about, being loudest at their location andquieter on devices further away. To an observer, this may be perceivedas though the media is originating at the user's location and the volumetrails off as distance increases, facilitating an immersive form ofvirtual surround or position-based audio.

FIG. 13 is a flow diagram illustrating an exemplary method 1000 forutilizing a PAN controller to operate a media computing device withmultiple virtual screens and personal area network capability as a homeautomation hub, according to a preferred embodiment of the invention. Inan initial step 1001, a media computing device 110 connects to a displaydevice 130 and begins drawing power for operation. In a next step 1002,a PAN controller 910 operating on media computing device 110 may beginoperation by connecting to IoT device over a personal area network. In anext step 1003, PAN controller 910 checks for known network addressesfor devices, and for any known devices it may connect directly using aunicast protocol 1004 a to immediately connect to the device and begincollecting information. PAN controller 910 may also request addressinformation and connections from any additional devices available on thenetwork using a broadcast protocol 1004 b, receiving device addressinformation as a response from any available devices. In a next step1005, PAN controller 910 receives device network addresses to maintainconnections as well as device capabilities for configuring operation andpresentation of devices to a user. Using the collected information, PANcontroller 910 may then produce an organized list of connected devices1006, and provide this list to a virtual screen driver 118 operating onmedia computing device 110 for presentation to a user via a virtualscreen presented on their device for interaction (as described above,referring to FIGS. 1-3).

FIG. 14 is a diagram illustrating an exemplary user interface 1400 forinteracting with PAN devices via a media computing device with multiplevirtual screens and personal area network capability, according to apreferred embodiment of the invention. According to the embodiment, whena media computing device 110 is operating as an IoT hub as describedabove (referring to FIG. 12), a user may view and interact withconnected PAN devices via a unified PAN interface 1410 shown on theirmobile device 120 a, as presented by virtual screen driver 118. WithinPAN interface 1410, a user may view and interact with connected PANdevices such as (for example) lights 1411 a-n or appliances 1412 a-n,for example to rename, reorganize, configure, or operate devices overthe network. According to the specific nature of a networkconfiguration, for example if a network is using the ZigBee protocol,devices may operate through a hub to provide their functionality in auser-addressable manner. According to the embodiment, a PAN controller910 may serve this role, so that no additional hardware is required andmedia computing device 110 may server the necessary functions of thedevice hub in addition to providing user-interactive means for operatingPAN devices directly.

Additionally, a user may be able to configure automated operation of PANdevices such as by configuring rules 1413 a-n, for example to directappliance to perform actions at specified times or lights to turn on orwhenever another action is detected on the network (for example, if amotion or door sensor detects the user has entered a room or building),or to configure devices to operate in response to conditions or actionon media computing device 110, such as to configure lights to operate inresponse to media being viewed. In this manner, using media computingdevice 110 as a PAN hub adds new functionality by combining the variousfunctions of a user's IoT devices directly with their media collectionand other functions of media computing device 110, without the need foradditional hardware or complicated technical setup. Additionally, thisenabled the configuration of complex rules and operation logic that maynot otherwise be possible by configuring PAN devices directly or byusing existing hub devices, by utilizing the more powerful storage andcomputing hardware of media computing device 110.

FIG. 15 is a flow diagram illustrating an exemplary method 1500 formultiuser interaction with PAN devices via a media computing device withmultiple virtual screens and personal area network capability, accordingto a preferred embodiment of the invention. In an initial step 1501, auser may connect to a virtual screen driver 118 operating on a mediacomputing device 110. In a next step 1502, virtual screen driver 118 maythen present the new user with a virtual screen corresponding to aunified PAN interface 1410, presenting available PAN devices for reviewor interaction. Upon receiving user interaction 1503 via the virtualscreen, a PAN controller 910 may check for a device cooldown 1504, forexample some network-connected light bulbs may have a built-in cooldowntimer to prevent rapid toggling of the light, which could pose a firehazard. If no device cooldown is active, PAN controller 910 directs thePAN device's operation 1505 based at least in part on a portion of theuser interaction received. If a cooldown is active, a notification maybe presented to the user 1507, such as a pop-up text notification ontheir device or an audible notification such as a tone or a voiceannouncement of the cooldown period remaining (for example), and maythen wait for the cooldown to expire before directing the PAN devicebased on the user's interaction 1508. In some arrangements, thisbehavior may be configurable such as to prompt a user to proceed orcancel the operation (for example, if they do not wish to wait for thecooldown to expire), or to configure the specific nature of anotification presented when a cooldown is encountered. Upon directing aPAN device, PAN controller may wait for or request an acknowledgment1506 from the PAN device that the operation was carried out, and maythen update the PAN interface 1509 to reflect the outcome of theoperation. In this manner, a user may interact with theirnetwork-connected devices such as smart home automation devicesincluding lights, appliances, and other device types, and may view allavailable devices in a single unified interface regardless of theirspecific functions, devices types, manufacturers or network protocols,and they may easily view and respond to the outcome of theirinteractions (such as if an operation fails, the PAN interface mayupdate to show the error information so the user can act appropriately).

FIG. 16 is a flow diagram illustrating an exemplary method 1600 forusing a media computing device with multiple virtual screens andpersonal area network capability as a mobile wireless access point. Inan initial step 1601, a media computing device 110 may connect to apower source, generally via a powered media hardware port such as HDMI105 (but it should be appreciated that in some arrangement, a power-onlyport or source may be used). Media computing device 110 may then scanfor local networks 1602 to connect to, for example networks known fromprevious connections or public network that allow connections withoutauthentication or configuration. If a suitable LAN is found, the devicemay then check for Internet connectivity 1603, to verify whetherexternal connectivity is possible on this network. If so, the device mayconnect to the LAN 1604 and begin operation.

However, if no LAN is found (or if a LAN is found but has no externalconnectivity), a media computing device 110 may then scan for anavailable WAN 1605 such as a cellular network that may be used. If a WANis available, the device may connect 1606 and begin broadcasting as anetwork access point 1607 and listening for user connections 1608 foroperation. In this manner, a user may connect to a media computingdevice 110 and utilize its functions without the need for a configuredLAN, such as when traveling or during periods of network loss. Thisenables the use of a media computing device 110 as an access point toallow WAN connectivity for user devices that may not have thatfunctionality, such as a laptop personal computer, as well as allowingfull network functionality such as streaming media and Internet browsingwhen no LAN is available. This also enables the use of media computingdevice 110 as a fallback network connection for a LAN, to continueoperation uninterrupted during network outages or hardware resets orfailures. These operation modes remove the need for separate and oftenexpensive devices to perform these roles, offering a user an“all-in-one” device for network, media, and IoT functionality.

If no WAN is available, a notification may be produced for presentationto a user (either upon connecting with their mobile device, so that uponconnecting they are immediately notified, or via an integralnotification mechanism such as an LED or hardware speaker) to alert themof limited connectivity, and may optionally offer limited functionality1610 such as for interacting with operating system features or viewinglocally-stored media, or other operation that may not require a networkconnection.

Hardware Architecture

Generally, the techniques disclosed herein may be implemented onhardware or a combination of software and hardware. For example, theymay be implemented in an operating system kernel, in a separate userprocess, in a library package bound into network applications, on aspecially constructed machine, on an application-specific integratedcircuit (ASIC), or on a network interface card.

Software/hardware hybrid implementations of at least some of theembodiments disclosed herein may be implemented on a programmablenetwork-resident machine (which should be understood to includeintermittently connected network-aware machines) selectively activatedor reconfigured by a computer program stored in memory. Such networkdevices may have multiple network interfaces that may be configured ordesigned to utilize different types of network communication protocols.A general architecture for some of these machines may be describedherein in order to illustrate one or more exemplary means by which agiven unit of functionality may be implemented. According to specificembodiments, at least some of the features or functionalities of thevarious embodiments disclosed herein may be implemented on one or moregeneral-purpose computers associated with one or more networks, such asfor example an end-user computer system, a client computer, a networkserver or other server system, a mobile computing device (e.g., tabletcomputing device, mobile phone, smartphone, laptop, or other appropriatecomputing device), a consumer electronic device, a music player, or anyother suitable electronic device, router, switch, or other suitabledevice, or any combination thereof. In at least some embodiments, atleast some of the features or functionalities of the various embodimentsdisclosed herein may be implemented in one or more virtualized computingenvironments (e.g., network computing clouds, virtual machines hosted onone or more physical computing machines, or other appropriate virtualenvironments).

Referring now to FIG. 7, there is shown a block diagram depicting anexemplary computing device 10 suitable for implementing at least aportion of the features or functionalities disclosed herein. Computingdevice 10 may be, for example, any one of the computing machines listedin the previous paragraph, or indeed any other electronic device capableof executing software- or hardware-based instructions according to oneor more programs stored in memory. Computing device 10 may be configuredto communicate with a plurality of other computing devices, such asclients or servers, over communications networks such as a wide areanetwork a metropolitan area network, a local area network, a wirelessnetwork, the Internet, or any other network, using known protocols forsuch communication, whether wireless or wired.

In one embodiment, computing device 10 includes one or more centralprocessing units (CPU) 12, one or more interfaces 15, and one or morebusses 14 (such as a peripheral component interconnect (PCI) bus). Whenacting under the control of appropriate software or firmware, CPU 12 maybe responsible for implementing specific functions associated with thefunctions of a specifically configured computing device or machine. Forexample, in at least one embodiment, a computing device 10 may beconfigured or designed to function as a server system utilizing CPU 12,local memory 11 and/or remote memory 16, and interface(s) 15. In atleast one embodiment, CPU 12 may be caused to perform one or more of thedifferent types of functions and/or operations under the control ofsoftware modules or components, which for example, may include anoperating system and any appropriate applications software, drivers, andthe like.

CPU 12 may include one or more processors 13 such as, for example, aprocessor from one of the Intel, ARM, Qualcomm, and AMD families ofmicroprocessors. In some embodiments, processors 13 may includespecially designed hardware such as application-specific integratedcircuits (ASICs), electrically erasable programmable read-only memories(EEPROMs), field-programmable gate arrays (FPGAs), and so forth, forcontrolling operations of computing device 10. In a specific embodiment,a local memory 11 (such as non-volatile random access memory (RAM)and/or read-only memory (ROM), including for example one or more levelsof cached memory) may also form part of CPU 12. However, there are manydifferent ways in which memory may be coupled to system 10. Memory 11may be used for a variety of purposes such as, for example, cachingand/or storing data, programming instructions, and the like. It shouldbe further appreciated that CPU 12 may be one of a variety ofsystem-on-a-chip (SOC) type hardware that may include additionalhardware such as memory or graphics processing chips, such as a QUALCOMMSNAPDRAGON™ or SAMSUNG EXYNOS™ CPU as are becoming increasingly commonin the art, such as for use in mobile devices or integrated devices.

As used herein, the term “processor” is not limited merely to thoseintegrated circuits referred to in the art as a processor, a mobileprocessor, or a microprocessor, but broadly refers to a microcontroller,a microcomputer, a programmable logic controller, anapplication-specific integrated circuit, and any other programmablecircuit.

In one embodiment, interfaces 15 are provided as network interface cards(NICs). Generally, NICs control the sending and receiving of datapackets over a computer network; other types of interfaces 15 may forexample support other peripherals used with computing device 10. Amongthe interfaces that may be provided are Ethernet interfaces, frame relayinterfaces, cable interfaces, DSL interfaces, token ring interfaces,graphics interfaces, and the like. In addition, various types ofinterfaces may be provided such as, for example, universal serial bus(USB), Serial, Ethernet, FIREWIRE™, THUNDERBOLT™, PCI, parallel, radiofrequency (RF), BLUETOOTH™, near-field communications (e.g., usingnear-field magnetics), 802.11 (WiFi), frame relay, TCP/IP, ISDN, fastEthernet interfaces, Gigabit Ethernet interfaces, Serial ATA (SATA) orexternal SATA (ESATA) interfaces, high-definition multimedia interface(HDMI), digital visual interface (DVI), analog or digital audiointerfaces, asynchronous transfer mode (ATM) interfaces, high-speedserial interface (HSSI) interfaces, Point of Sale (POS) interfaces,fiber data distributed interfaces (FDDIs), and the like. Generally, suchinterfaces 15 may include physical ports appropriate for communicationwith appropriate media. In some cases, they may also include anindependent processor (such as a dedicated audio or video processor, asis common in the art for high-fidelity A/V hardware interfaces) and, insome instances, volatile and/or non-volatile memory (e.g., RAM).

Although the system shown in FIG. 7 illustrates one specificarchitecture for a computing device 10 for implementing one or more ofthe inventions described herein, it is by no means the only devicearchitecture on which at least a portion of the features and techniquesdescribed herein may be implemented. For example, architectures havingone or any number of processors 13 may be used, and such processors 13may be present in a single device or distributed among any number ofdevices. In one embodiment, a single processor 13 handles communicationsas well as routing computations, while in other embodiments a separatededicated communications processor may be provided. In variousembodiments, different types of features or functionalities may beimplemented in a system according to the invention that includes aclient device (such as a tablet device or smartphone running clientsoftware) and server systems (such as a server system described in moredetail below).

Regardless of network device configuration, the system of the presentinvention may employ one or more memories or memory modules (such as,for example, remote memory block 16 and local memory 11) configured tostore data, program instructions for the general-purpose networkoperations, or other information relating to the functionality of theembodiments described herein (or any combinations of the above). Programinstructions may control execution of or comprise an operating systemand/or one or more applications, for example. Memory 16 or memories 11,16 may also be configured to store data structures, configuration data,encryption data, historical system operations information, or any otherspecific or generic non-program information described herein.

Because such information and program instructions may be employed toimplement one or more systems or methods described herein, at least somenetwork device embodiments may include nontransitory machine-readablestorage media, which, for example, may be configured or designed tostore program instructions, state information, and the like forperforming various operations described herein. Examples of suchnontransitory machine-readable storage media include, but are notlimited to, magnetic media such as hard disks, floppy disks, andmagnetic tape; optical media such as CD-ROM disks; magneto-optical mediasuch as optical disks, and hardware devices that are speciallyconfigured to store and perform program instructions, such as read-onlymemory devices (ROM), flash memory (as is common in mobile devices andintegrated systems), solid state drives (SSD) and “hybrid SSD” storagedrives that may combine physical components of solid state and hard diskdrives in a single hardware device (as are becoming increasingly commonin the art with regard to personal computers), memristor memory, randomaccess memory (RAM), and the like. It should be appreciated that suchstorage means may be integral and non-removable (such as RAM hardwaremodules that may be soldered onto a motherboard or otherwise integratedinto an electronic device), or they may be removable such as swappableflash memory modules (such as “thumb drives” or other removable mediadesigned for rapidly exchanging physical storage devices),“hot-swappable” hard disk drives or solid state drives, removableoptical storage discs, or other such removable media, and that suchintegral and removable storage media may be utilized interchangeably.Examples of program instructions include both object code, such as maybe produced by a compiler, machine code, such as may be produced by anassembler or a linker, byte code, such as may be generated by forexample a JAVA™ compiler and may be executed using a Java virtualmachine or equivalent, or files containing higher level code that may beexecuted by the computer using an interpreter (for example, scriptswritten in Python, Perl, Ruby, Groovy, or any other scripting language).

In some embodiments, systems according to the present invention may beimplemented on a standalone computing system. Referring now to FIG. 8,there is shown a block diagram depicting a typical exemplaryarchitecture of one or more embodiments or components thereof on astandalone computing system. Computing device 20 includes processors 21that may run software that carry out one or more functions orapplications of embodiments of the invention, such as for example aclient application 24. Processors 21 may carry out computinginstructions under control of an operating system 22 such as, forexample, a version of MICROSOFT WINDOWS™ operating system, APPLE OSX™ oriOS™ operating systems, some variety of the Linux operating system,ANDROID™ operating system, or the like. In many cases, one or moreshared services 23 may be operable in system 20, and may be useful forproviding common services to client applications 24. Services 23 may forexample be WINDOWS™ services, user-space common services in a Linuxenvironment, or any other type of common service architecture used withoperating system 21. Input devices 28 may be of any type suitable forreceiving user input, including for example a keyboard, touchscreen,microphone (for example, for voice input), mouse, touchpad, trackball,or any combination thereof. Output devices 27 may be of any typesuitable for providing output to one or more users, whether remote orlocal to system 20, and may include for example one or more screens forvisual output, speakers, printers, or any combination thereof. Memory 25may be random-access memory having any structure and architecture knownin the art, for use by processors 21, for example to run software.Storage devices 26 may be any magnetic, optical, mechanical, memristor,or electrical storage device for storage of data in digital form (suchas those described above, referring to FIG. 7). Examples of storagedevices 26 include flash memory, magnetic hard drive, CD-ROM, and/or thelike.

In some embodiments, systems of the present invention may be implementedon a distributed computing network, such as one having any number ofclients and/or servers. Referring now to FIG. 9, there is shown a blockdiagram depicting an exemplary architecture 30 for implementing at leasta portion of a system according to an embodiment of the invention on adistributed computing network. According to the embodiment, any numberof clients 33 may be provided. Each client 33 may run software forimplementing client-side portions of the present invention; clients maycomprise a system 20 such as that illustrated in FIG. 8. In addition,any number of servers 32 may be provided for handling requests receivedfrom one or more clients 33. Clients 33 and servers 32 may communicatewith one another via one or more electronic networks 31, which may be invarious embodiments any of the Internet, a wide area network, a mobiletelephony network (such as CDMA or GSM cellular networks), a wirelessnetwork (such as WiFi, WiMAX, LTE, and so forth), or a local areanetwork (or indeed any network topology known in the art; the inventiondoes not prefer any one network topology over any other). Networks 31may be implemented using any known network protocols, including forexample wired and/or wireless protocols.

In addition, in some embodiments, servers 32 may call external services37 when needed to obtain additional information, or to refer toadditional data concerning a particular call. Communications withexternal services 37 may take place, for example, via one or morenetworks 31. In various embodiments, external services 37 may compriseweb-enabled services or functionality related to or installed on thehardware device itself. For example, in an embodiment where clientapplications 24 are implemented on a smartphone or other electronicdevice, client applications 24 may obtain information stored in a serversystem 32 in the cloud or on an external service 37 deployed on one ormore of a particular enterprise's or user's premises.

In some embodiments of the invention, clients 33 or servers 32 (or both)may make use of one or more specialized services or appliances that maybe deployed locally or remotely across one or more networks 31. Forexample, one or more databases 34 may be used or referred to by one ormore embodiments of the invention. It should be understood by one havingordinary skill in the art that databases 34 may be arranged in a widevariety of architectures and using a wide variety of data access andmanipulation means. For example, in various embodiments one or moredatabases 34 may comprise a relational database system using astructured query language (SQL), while others may comprise analternative data storage technology such as those referred to in the artas “NoSQL” (for example, HADOOP CASSANDRA™, GOOGLE BIGTABLE™, and soforth). In some embodiments, variant database architectures such ascolumn-oriented databases, in-memory databases, clustered databases,distributed databases, or even flat file data repositories may be usedaccording to the invention. It will be appreciated by one havingordinary skill in the art that any combination of known or futuredatabase technologies may be used as appropriate, unless a specificdatabase technology or a specific arrangement of components is specifiedfor a particular embodiment herein. Moreover, it should be appreciatedthat the term “database” as used herein may refer to a physical databasemachine, a cluster of machines acting as a single database system, or alogical database within an overall database management system. Unless aspecific meaning is specified for a given use of the term “database”, itshould be construed to mean any of these senses of the word, all ofwhich are understood as a plain meaning of the term “database” by thosehaving ordinary skill in the art.

Similarly, most embodiments of the invention may make use of one or moresecurity systems 36 and configuration systems 35. Security andconfiguration management are common information technology (IT) and webfunctions, and some amount of each are generally associated with any ITor web systems. It should be understood by one having ordinary skill inthe art that any configuration or security subsystems known in the artnow or in the future may be used in conjunction with embodiments of theinvention without limitation, unless a specific security 36 orconfiguration system 35 or approach is specifically required by thedescription of any specific embodiment.

FIG. 10 shows an exemplary overview of a computer system 40 as may beused in any of the various locations throughout the system. It isexemplary of any computer that may execute code to process data. Variousmodifications and changes may be made to computer system 40 withoutdeparting from the broader scope of the system and method disclosedherein. Central processor unit (CPU) 41 is connected to bus 42, to whichbus is also connected memory 43, nonvolatile memory 44, display 47,input/output (I/O) unit 48, and network interface card (NIC) 53. I/Ounit 48 may, typically, be connected to keyboard 49, pointing device 50,hard disk 52, and real-time clock 51. NIC 53 connects to network 54,which may be the Internet or a local network, which local network may ormay not have connections to the Internet. Also shown as part of system40 is power supply unit 45 connected, in this example, to a mainalternating current (AC) supply 46. Not shown are batteries that couldbe present, and many other devices and modifications that are well knownbut are not applicable to the specific novel functions of the currentsystem and method disclosed herein. It should be appreciated that someor all components illustrated may be combined, such as in variousintegrated applications, for example Qualcomm or Samsungsystem-on-a-chip (SOC) devices, or whenever it may be appropriate tocombine multiple capabilities or functions into a single hardware device(for instance, in mobile devices such as smartphones, video gameconsoles, in-vehicle computer systems such as navigation or multimediasystems in automobiles, or other integrated hardware devices).

In various embodiments, functionality for implementing systems ormethods of the present invention may be distributed among any number ofclient and/or server components. For example, various software modulesmay be implemented for performing various functions in connection withthe present invention, and such modules may be variously implemented torun on server and/or client components.

The skilled person will be aware of a range of possible modifications ofthe various embodiments described above. Accordingly, the presentinvention is defined by the claims and their equivalents.

1. A system for multiuser interaction with a media computing device withmultiple virtual screens, the system comprising: a media computingdevice comprising a processor, a memory, a high-definition multimediainterface adapter, a wireless network interface, a virtual screendriver, and an operating system; and a plurality of mobile devices, eachcomprising at least a processor, a memory, a touchscreen display, and awireless network interface; wherein the media computing device sendsscreen graphics using the virtual screen driver to the plurality ofmobile devices using the wireless network, providing thereby a complete,independent logical operating system interface to each mobile device;wherein each of the plurality of mobile devices displays a virtualscreen of the operating system using the screen graphics sent by themedia computing device; wherein touchscreen user interaction events aretransmitted from the plurality of mobile devices to the media computingdevice and processed by the operating system thereof.
 2. A method formanaging a multiplayer video game using multiuser interaction with amedia computing device with multiple virtual screens, comprising thesteps of: running a game application in user space of an operatingsystem of a media computing device; sending, using a high-definitionmultimedia interface adapter of the media computing device, an overviewgame display for a game application to a high-definition multimediadisplay device physically connected to the high-definition multimediainterface adapter of the media computing device; presenting, usingscreen graphics generated by a virtual screen driver of the mediacomputing device, a virtual screen comprising a player-specific gamedisplay on each of a plurality of connected mobile devices, providingthereby a complete, independent logical user interface to each mobiledevice; receiving a user interaction event from a first mobile device atthe media computing device; processing, using the game application, theuser interaction event to determine whether each respectiveplayer-specific game display or the overview game display is affected bythe user interaction event; and updating the affected game displays. 3.A method for handling multi-user audio using multiuser interaction witha media computing device, comprising the steps of: transmitting screengraphics generated by a virtual screen driver of the media computingdevice, the screen graphics corresponding to a virtual screen comprisinga media control interface, to a plurality of connected mobile devices,providing thereby a complete, independent logical media controlinterface to each mobile device; receiving, at the media computingdevice, a plurality of user interaction events from the plurality ofmobile devices; selecting a plurality of audio playback devices based ona first user interaction event; selecting media content for playbackbased on a second user interaction event; and wirelessly transmitting,from the media computing device, audio from the selected media contentfor playback at the plurality of audio playback devices.
 4. The methodof claim 3, further comprising the steps of: sending, using ahigh-definition multimedia interface adapter of a media computingdevice, high-definition video and a plurality of audio channelsassociated with the media content to a high-definition multimediadisplay device physically connected to the high-definition multimediainterface adapter of the media computing device.
 5. The method of claim4, wherein playback of the video and audio sent to the high-definitionmultimedia display device is synchronized with playback of the audiotransmitted to the plurality of audio playback devices.